[pull] main from apache:main by pull[bot] · Pull Request #1337 · TCeason/arrow-datafusion

pull · 2026-07-14T16:40:13Z

See Commits and Changes for more details.

Created by pull[bot] (v2.0.0-alpha.4)

Can you help keep this open source service alive? 💖 Please sponsor : )

## Which issue does this PR close? - Closes #23351. ## Rationale for this change Benches were added recently in #23350 . Improve and measure some optimisations. ## What changes are included in this PR? - Improve performance for `seconds_ns` and `seconds_as_i32`. A pretty rare code path. ## Are these changes tested? - Tests are passing - Bench-measured perf is improving ## Are there any user-facing changes?   --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>

## Which issue does this PR close? N/A ## Rationale for this change Optimize an existing function ## What changes are included in this PR? round() over a no-null float column with scalar decimal_places now hoists the 10^dp factor out of the loop and uses the vectorizable infallible `unary` kernel instead of per-element `try_unary`+`powi`, with a fallback preserving bit-identical output for null/array cases. ## Are these changes tested? Existing tests Benchmark (criterion): - 8192: 28.689% faster (base 2711ns -> cand 1933ns) - 4096: 41.544% faster (base 1382ns -> cand 808ns) - 1024: 25.253% faster (base 451ns -> cand 337ns) - 8192: 44.103% faster (base 1486ns -> cand 831ns) - 4096: 29.01% faster (base 679ns -> cand 482ns) - 1024: 20.457% faster (base 291ns -> cand 231ns) ## Are there any user-facing changes? No

## Which issue does this PR close? N/A ## Rationale for this change Optimize existing expression. ## What changes are included in this PR? Build the trimmed Utf8/LargeUtf8 output by writing slices into a single exactly-sized value buffer and reusing the input null buffer, instead of collecting through arrow's FromIterator, which starts its value buffer at 1024 bytes, grows it by repeated doubling, and rebuilds the null bitmap row by row. ## Are these changes tested? Existing tests. Benchmark (criterion): - ltrim string_view [size=4096, len=64, remaining=60]: -2.285% faster (base 51968ns -> cand 53155ns) - ltrim large_string [size=4096, len=64, remaining=60]: 10.791% faster (base 70426ns -> cand 62827ns) - ltrim string [size=4096, len=64, remaining=60]: 15.055% faster (base 71344ns -> cand 60603ns) - btrim string_view [size=4096, len=64, remaining=60]: 0.182% faster (base 87918ns -> cand 87758ns) - btrim large_string [size=4096, len=64, remaining=60]: 9.316% faster (base 107290ns -> cand 97295ns) - rtrim large_string [size=4096, len=64, remaining=60]: 12.753% faster (base 72178ns -> cand 62973ns) - btrim string [size=4096, len=64, remaining=60]: 9.067% faster (base 105930ns -> cand 96326ns) - rtrim string_view [size=4096, len=64, remaining=60]: 2.012% faster (base 52285ns -> cand 51233ns) - rtrim string [size=4096, len=64, remaining=60]: 11.4% faster (base 69425ns -> cand 61511ns) - btrim string [size=4096, len=64, remaining=8]: 4.755% faster (base 803627ns -> cand 765412ns) - btrim large_string [size=4096, len=64, remaining=8]: 4.711% faster (base 802769ns -> cand 764954ns) - rtrim string_view [size=4096, len=64, remaining=8]: 3.178% faster (base 416499ns -> cand 403262ns) - ltrim string_view [size=4096, len=64, remaining=8]: -0.766% faster (base 377205ns -> cand 380095ns) - ltrim large_string [size=4096, len=64, remaining=8]: 3.418% faster (base 385659ns -> cand 372477ns) - btrim string_view [size=4096, len=64, remaining=8]: -1.094% faster (base 763164ns -> cand 771516ns) - ltrim string [size=4096, len=64, remaining=8]: 2.05% faster (base 380670ns -> cand 372867ns) - rtrim large_string [size=4096, len=64, remaining=8]: 1.891% faster (base 412638ns -> cand 404835ns) - rtrim string [size=4096, len=64, remaining=8]: -0.213% faster (base 407115ns -> cand 407983ns) - ltrim string [size=4096, len=12, pad=4]: 15.663% faster (base 24752ns -> cand 20875ns) - rtrim string [size=4096, len=12, pad=4]: 19.084% faster (base 25496ns -> cand 20631ns) - rtrim string_view [size=4096, len=12, pad=4]: 0.935% faster (base 23452ns -> cand 23232ns) - btrim large_string [size=4096, len=16, pad=4]: 22.842% faster (base 36829ns -> cand 28417ns) - rtrim large_string [size=4096, len=12, pad=4]: 19.476% faster (base 26289ns -> cand 21169ns) - btrim string [size=4096, len=16, pad=4]: 21.121% faster (base 36734ns -> cand 28976ns) - ltrim string_view [size=4096, len=12, pad=4]: -0.663% faster (base 22858ns -> cand 23010ns) - ltrim large_string [size=4096, len=12, pad=4]: 18.163% faster (base 25103ns -> cand 20543ns) - btrim string_view [size=4096, len=16, pad=4]: 1.32% faster (base 34615ns -> cand 34158ns) - btrim string_view [size=4096, len=68, pad=4]: 2.079% faster (base 33597ns -> cand 32898ns) - ltrim string_view [size=4096, len=64, pad=4]: -2.779% faster (base 21991ns -> cand 22602ns) - btrim string [size=4096, len=68, pad=4]: 15.112% faster (base 48987ns -> cand 41584ns) - ltrim large_string [size=4096, len=64, pad=4]: 16.025% faster (base 41640ns -> cand 34967ns) - btrim large_string [size=4096, len=68, pad=4]: 17.146% faster (base 49353ns -> cand 40891ns) - rtrim string [size=4096, len=64, pad=4]: 21.115% faster (base 42069ns -> cand 33186ns) - ltrim string [size=4096, len=64, pad=4]: 19.147% faster (base 41375ns -> cand 33453ns) - rtrim string_view [size=4096, len=64, pad=4]: -1.255% faster (base 22190ns -> cand 22468ns) - rtrim large_string [size=4096, len=64, pad=4]: 23.5% faster (base 44456ns -> cand 34008ns) - rtrim large_string [size=4096, len=12, remaining=8]: 14.652% faster (base 55987ns -> cand 47784ns) - rtrim string [size=4096, len=12, remaining=8]: 15.777% faster (base 55511ns -> cand 46753ns) - btrim string_view [size=4096, len=12, remaining=8]: 0.582% faster (base 90871ns -> cand 90343ns) - ltrim string [size=4096, len=12, remaining=8]: 15.751% faster (base 55374ns -> cand 46652ns) - ltrim string_view [size=4096, len=12, remaining=8]: 1.227% faster (base 70019ns -> cand 69161ns) - rtrim string_view [size=4096, len=12, remaining=8]: 2.434% faster (base 54018ns -> cand 52703ns) - ltrim large_string [size=4096, len=12, remaining=8]: 15.63% faster (base 56321ns -> cand 47518ns) - btrim string [size=4096, len=12, remaining=8]: 29.843% faster (base 119405ns -> cand 83771ns) - btrim large_string [size=4096, len=12, remaining=8]: 29.726% faster (base 119617ns -> cand 84060ns) - btrim string [size=4096, len=120, pad=56]: 5.423% faster (base 134295ns -> cand 127012ns) - btrim large_string [size=4096, len=120, pad=56]: 4.731% faster (base 135661ns -> cand 129243ns) - ltrim large_string [size=4096, len=64, pad=56]: 10.671% faster (base 78904ns -> cand 70483ns) - ltrim string_view [size=4096, len=64, pad=56]: 0.467% faster (base 76297ns -> cand 75940ns) - rtrim large_string [size=4096, len=64, pad=56]: 10.909% faster (base 79353ns -> cand 70696ns) - rtrim string [size=4096, len=64, pad=56]: 11.069% faster (base 78551ns -> cand 69856ns) - btrim string_view [size=4096, len=120, pad=56]: 0.581% faster (base 133912ns -> cand 133135ns) - rtrim string_view [size=4096, len=64, pad=56]: -0.131% faster (base 78208ns -> cand 78310ns) - ltrim string [size=4096, len=64, pad=56]: 9.939% faster (base 78138ns -> cand 70372ns) Full criterion output: ```text short strings (len <= 12)/ltrim string_view [size=4096, len=12, remaining=8] time: [68.772 µs 69.161 µs 69.588 µs] change: [−2.1338% −1.2266% −0.3627%] (p = 0.02 < 0.05) Change within noise threshold. short strings (len <= 12)/ltrim string [size=4096, len=12, remaining=8] time: [46.637 µs 46.653 µs 46.672 µs] change: [−16.261% −15.751% −15.300%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 10 measurements (20.00%) 1 (10.00%) low mild 1 (10.00%) high severe short strings (len <= 12)/ltrim large_string [size=4096, len=12, remaining=8] time: [47.002 µs 47.519 µs 48.116 µs] change: [−17.005% −15.630% −14.316%] (p = 0.00 < 0.05) Performance has improved. Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) high mild long strings, short trim/ltrim string_view [size=4096, len=64, remaining=60] time: [52.566 µs 53.156 µs 53.782 µs] change: [+1.2396% +2.2849% +3.6410%] (p = 0.00 < 0.05) Performance has regressed. long strings, short trim/ltrim string [size=4096, len=64, remaining=60] time: [59.864 µs 60.603 µs 61.344 µs] change: [−16.396% −15.055% −13.779%] (p = 0.00 < 0.05) Performance has improved. long strings, short trim/ltrim large_string [size=4096, len=64, remaining=60] time: [62.154 µs 62.827 µs 63.567 µs] change: [−11.916% −10.791% −9.7385%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 10 measurements (20.00%) 2 (20.00%) high mild long strings, long trim/ltrim string_view [size=4096, len=64, remaining=8] time: [377.82 µs 380.10 µs 382.80 µs] change: [+0.1236% +0.7661% +1.5272%] (p = 0.02 < 0.05) Change within noise threshold. Found 2 outliers among 10 measurements (20.00%) 2 (20.00%) high mild long strings, long trim/ltrim string [size=4096, len=64, remaining=8] time: [372.10 µs 372.87 µs 373.80 µs] change: [−2.2654% −2.0500% −1.7991%] (p = 0.00 < 0.05) Performance has improved. Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) high mild long strings, long trim/ltrim large_string [size=4096, len=64, remaining=8] time: [372.15 µs 372.48 µs 373.07 µs] change: [−4.2678% −3.4179% −2.6627%] (p = 0.00 < 0.05) Performance has improved. Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) high severe trim spaces, short strings (len <= 12)/ltrim string_view [size=4096, len=12, pad=4] time: [22.970 µs 23.010 µs 23.049 µs] change: [+0.3046% +0.6633% +1.0373%] (p = 0.00 < 0.05) Change within noise threshold. trim spaces, short strings (len <= 12)/ltrim string [size=4096, len=12, pad=4] time: [20.827 µs 20.876 µs 20.938 µs] change: [−15.861% −15.663% −15.422%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 10 measurements (20.00%) 1 (10.00%) high mild 1 (10.00%) high severe trim spaces, short strings (len <= 12)/ltrim large_string [size=4096, len=12, pad=4] time: [20.530 µs 20.544 µs 20.561 µs] change: [−18.501% −18.163% −17.924%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 10 measurements (20.00%) 2 (20.00%) high mild trim spaces, long strings/ltrim string_view [size=4096, len=64, pad=4] time: [22.593 µs 22.603 µs 22.613 µs] change: [+2.5048% +2.7791% +3.0854%] (p = 0.00 < 0.05) Performance has regressed. trim spaces, long strings/ltrim string [size=4096, len=64, pad=4] time: [33.443 µs 33.454 µs 33.471 µs] change: [−19.949% −19.147% −18.364%] (p = 0.00 < 0.05) Performance has improved. Found 3 outliers among 10 measurements (30.00%) 1 (10.00%) low mild 2 (20.00%) high severe trim spaces, long strings/ltrim large_string [size=4096, len=64, pad=4] time: [34.630 µs 34.967 µs 35.563 µs] change: [−18.222% −16.025% −13.920%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 10 measurements (20.00%) 2 (20.00%) high severe trim spaces, heavy padding/ltrim string_view [size=4096, len=64, pad=56] time: [75.906 µs 75.941 µs 76.000 µs] change: [−1.1903% −0.4672% +0.0722%] (p = 0.19 > 0.05) No change in performance detected. Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) high severe trim spaces, heavy padding/ltrim string [size=4096, len=64, pad=56] time: [70.315 µs 70.372 µs 70.433 µs] change: [−10.488% −9.9391% −9.3406%] (p = 0.00 < 0.05) Performance has improved. trim spaces, heavy padding/ltrim large_string [size=4096, len=64, pad=56] time: [70.440 µs 70.484 µs 70.545 µs] change: [−11.233% −10.671% −10.167%] (p = 0.00 < 0.05) Performance has improved. Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) high mild short strings (len <= 12)/rtrim string_view [size=4096, len=12, remaining=8] time: [52.475 µs 52.704 µs 52.943 µs] change: [−3.0207% −2.4340% −1.8783%] (p = 0.00 < 0.05) Performance has improved. short strings (len <= 12)/rtrim string [size=4096, len=12, remaining=8] time: [46.561 µs 46.754 µs 46.949 µs] change: [−16.647% −15.777% −14.958%] (p = 0.00 < 0.05) Performance has improved. short strings (len <= 12)/rtrim large_string [size=4096, len=12, remaining=8] time: [47.477 µs 47.784 µs 48.152 µs] change: [−15.255% −14.652% −14.023%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 10 measurements (20.00%) 2 (20.00%) high mild long strings, short trim/rtrim string_view [size=4096, len=64, remaining=60] time: [51.031 µs 51.233 µs 51.459 µs] change: [−2.4596% −2.0121% −1.5299%] (p = 0.00 < 0.05) Performance has improved. long strings, short trim/rtrim string [size=4096, len=64, remaining=60] time: [60.549 µs 61.511 µs 62.451 µs] change: [−13.110% −11.400% −9.7295%] (p = 0.00 < 0.05) Performance has improved. long strings, short trim/rtrim large_string [size=4096, len=64, remaining=60] time: [62.151 µs 62.974 µs 63.706 µs] change: [−14.250% −12.753% −11.311%] (p = 0.00 < 0.05) Performance has improved. long strings, long trim/rtrim string_view [size=4096, len=64, remaining=8] time: [400.34 µs 403.26 µs 407.94 µs] change: [−4.9423% −3.1783% −1.3341%] (p = 0.01 < 0.05) Performance has improved. Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) high severe long strings, long trim/rtrim string [size=4096, len=64, remaining=8] time: [404.95 µs 407.98 µs 411.21 µs] change: [−1.6939% +0.2132% +1.9796%] (p = 0.83 > 0.05) No change in performance detected. long strings, long trim/rtrim large_string [size=4096, len=64, remaining=8] time: [401.16 µs 404.84 µs 409.18 µs] change: [−3.2274% −1.8912% −0.5131%] (p = 0.02 < 0.05) Change within noise threshold. Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) high mild trim spaces, short strings (len <= 12)/rtrim string_view [size=4096, len=12, pad=4] time: [23.042 µs 23.233 µs 23.388 µs] change: [−1.8636% −0.9354% −0.1975%] (p = 0.04 < 0.05) Change within noise threshold. Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) low mild trim spaces, short strings (len <= 12)/rtrim string [size=4096, len=12, pad=4] time: [20.621 µs 20.631 µs 20.645 µs] change: [−19.407% −19.084% −18.841%] (p = 0.00 < 0.05) Performance has improved. Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) high mild trim spaces, short strings (len <= 12)/rtrim large_string [size=4096, len=12, pad=4] time: [21.157 µs 21.170 µs 21.185 µs] change: [−20.204% −19.476% −18.798%] (p = 0.00 < 0.05) Performance has improved. Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) high mild trim spaces, long strings/rtrim string_view [size=4096, len=64, pad=4] time: [22.455 µs 22.469 µs 22.485 µs] change: [+0.6989% +1.2553% +1.7874%] (p = 0.00 < 0.05) Change within noise threshold. Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) high mild trim spaces, long strings/rtrim string [size=4096, len=64, pad=4] time: [33.171 µs 33.186 µs 33.211 µs] change: [−21.690% −21.115% −20.557%] (p = 0.00 < 0.05) Performance has improved. Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) high severe trim spaces, long strings/rtrim large_string [size=4096, len=64, pad=4] time: [33.869 µs 34.009 µs 34.146 µs] change: [−24.131% −23.500% −22.869%] (p = 0.00 < 0.05) Performance has improved. trim spaces, heavy padding/rtrim string_view [size=4096, len=64, pad=56] time: [78.197 µs 78.310 µs 78.440 µs] change: [−0.7780% +0.1305% +1.0420%] (p = 0.79 > 0.05) No change in performance detected. trim spaces, heavy padding/rtrim string [size=4096, len=64, pad=56] time: [69.823 µs 69.857 µs 69.907 µs] change: [−11.397% −11.069% −10.765%] (p = 0.00 < 0.05) Performance has improved. Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) high severe trim spaces, heavy padding/rtrim large_string [size=4096, len=64, pad=56] time: [70.631 µs 70.696 µs 70.773 µs] change: [−11.374% −10.909% −10.464%] (p = 0.00 < 0.05) Performance has improved. short strings (len <= 12)/btrim string_view [size=4096, len=12, remaining=8] time: [89.699 µs 90.343 µs 91.130 µs] change: [−1.8718% −0.5816% +0.4785%] (p = 0.42 > 0.05) No change in performance detected. Found 2 outliers among 10 measurements (20.00%) 1 (10.00%) high mild 1 (10.00%) high severe short strings (len <= 12)/btrim string [size=4096, len=12, remaining=8] time: [83.398 µs 83.771 µs 84.145 µs] change: [−30.515% −29.843% −29.167%] (p = 0.00 < 0.05) Performance has improved. short strings (len <= 12)/btrim large_string [size=4096, len=12, remaining=8] time: [83.788 µs 84.060 µs 84.366 µs] change: [−29.968% −29.726% −29.499%] (p = 0.00 < 0.05) Performance has improved. long strings, short trim/btrim string_view [size=4096, len=64, remaining=60] time: [87.545 µs 87.758 µs 88.028 µs] change: [−0.5010% −0.1823% +0.1714%] (p = 0.34 > 0.05) No change in performance detected. Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) high mild long strings, short trim/btrim string [size=4096, len=64, remaining=60] time: [96.102 µs 96.326 µs 96.558 µs] change: [−9.3462% −9.0668% −8.7659%] (p = 0.00 < 0.05) Performance has improved. long strings, short trim/btrim large_string [size=4096, len=64, remaining=60] time: [97.185 µs 97.296 µs 97.406 µs] change: [−9.6125% −9.3157% −9.0282%] (p = 0.00 < 0.05) Performance has improved. long strings, long trim/btrim string_view [size=4096, len=64, remaining=8] time: [770.37 µs 771.52 µs 772.90 µs] change: [+0.6693% +1.0943% +1.4767%] (p = 0.00 < 0.05) Change within noise threshold. Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) high mild long strings, long trim/btrim string [size=4096, len=64, remaining=8] time: [763.92 µs 765.41 µs 766.99 µs] change: [−5.0952% −4.7554% −4.4481%] (p = 0.00 < 0.05) Performance has improved. long strings, long trim/btrim large_string [size=4096, len=64, remaining=8] time: [764.37 µs 764.95 µs 765.72 µs] change: [−4.8509% −4.7107% −4.5654%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 10 measurements (20.00%) 2 (20.00%) high severe trim spaces, short strings (len <= 12)/btrim string_view [size=4096, len=16, pad=4] time: [34.141 µs 34.159 µs 34.179 µs] change: [−1.3904% −1.3204% −1.2474%] (p = 0.00 < 0.05) Performance has improved. Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) high mild trim spaces, short strings (len <= 12)/btrim string [size=4096, len=16, pad=4] time: [28.965 µs 28.976 µs 28.990 µs] change: [−21.262% −21.121% −20.969%] (p = 0.00 < 0.05) Performance has improved. Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) high mild trim spaces, short strings (len <= 12)/btrim large_string [size=4096, len=16, pad=4] time: [28.235 µs 28.417 µs 28.634 µs] change: [−23.842% −22.842% −21.895%] (p = 0.00 < 0.05) Performance has improved. Found 3 outliers among 10 measurements (30.00%) 1 (10.00%) low mild 2 (20.00%) high severe trim spaces, long strings/btrim string_view [size=4096, len=68, pad=4] time: [32.686 µs 32.899 µs 33.100 µs] change: [−3.4430% −2.0786% −0.9461%] (p = 0.00 < 0.05) Change within noise threshold. trim spaces, long strings/btrim string [size=4096, len=68, pad=4] time: [41.553 µs 41.585 µs 41.622 µs] change: [−15.330% −15.112% −14.914%] (p = 0.00 < 0.05) Performance has improved. Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) high mild trim spaces, long strings/btrim large_string [size=4096, len=68, pad=4] time: [40.584 µs 40.891 µs 41.236 µs] change: [−17.802% −17.146% −16.404%] (p = 0.00 < 0.05) Performance has improved. trim spaces, heavy padding/btrim string_view [size=4096, len=120, pad=56] time: [133.07 µs 133.14 µs 133.23 µs] change: [−0.6635% −0.5806% −0.4984%] (p = 0.00 < 0.05) Change within noise threshold. Found 2 outliers among 10 measurements (20.00%) 2 (20.00%) high severe trim spaces, heavy padding/btrim string [size=4096, len=120, pad=56] time: [126.90 µs 127.01 µs 127.15 µs] change: [−5.5277% −5.4230% −5.2985%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 10 measurements (20.00%) 1 (10.00%) high mild 1 (10.00%) high severe trim spaces, heavy padding/btrim large_string [size=4096, len=120, pad=56] time: [128.87 µs 129.24 µs 129.72 µs] change: [−5.0133% −4.7314% −4.3817%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 10 measurements (20.00%) 1 (10.00%) high mild 1 (10.00%) high severe ``` ## Are there any user-facing changes? No

…awn_buffered` (#23560) ## Which issue does this PR close? Fixes: #23559 ## Rationale for this change See the issue description ## What changes are included in this PR? Change spawn_buffered's behavior to ensure a maximum of `buffer` RecordBatches are produced before any consumer polls the stream. ## Are these changes tested? Yes ## Are there any user-facing changes? No

## Which issue does this PR close? N/A ## Rationale for this change Optimize existing expression. ## What changes are included in this PR? Replaced the per-row chrono NaiveDateTime round trip in the untimezoned coarse path of date_trunc (week/month/quarter/year) with integer civil-calendar arithmetic, cutting several allocationless-but-validating chrono field setters down to a few integer ops per row. ## Are these changes tested? Existing tests. Benchmark (criterion): - date_trunc_month_nanos_1000: 48.452% faster (base 19164ns -> cand 9878ns) - date_trunc_week_nanos_1000: 76.045% faster (base 23337ns -> cand 5590ns) - date_trunc_month_second_1000: 47.688% faster (base 17376ns -> cand 9090ns) - date_trunc_quarter_nanos_1000: 32.493% faster (base 22005ns -> cand 14854ns) - date_trunc_year_nanos_1000: 39.449% faster (base 20460ns -> cand 12388ns) - date_trunc_minute_1000: 1.2% faster (base 654ns -> cand 647ns) Full criterion output: ```text date_trunc_minute_1000 time: [645.50 ns 652.66 ns 659.52 ns] change: [−2.0837% −1.1996% −0.2678%] (p = 0.01 < 0.05) Change within noise threshold. Found 3 outliers among 100 measurements (3.00%) 3 (3.00%) high mild date_trunc_month_second_1000 time: [9.0984 µs 9.1139 µs 9.1298 µs] change: [−47.773% −47.688% −47.597%] (p = 0.00 < 0.05) Performance has improved. Found 1 outliers among 100 measurements (1.00%) 1 (1.00%) high severe date_trunc_week_nanos_1000 time: [5.5882 µs 5.6103 µs 5.6308 µs] change: [−76.154% −76.045% −75.951%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 100 measurements (2.00%) 2 (2.00%) low mild date_trunc_month_nanos_1000 time: [9.8640 µs 9.8831 µs 9.9059 µs] change: [−48.576% −48.452% −48.309%] (p = 0.00 < 0.05) Performance has improved. Found 4 outliers among 100 measurements (4.00%) 3 (3.00%) high mild 1 (1.00%) high severe date_trunc_quarter_nanos_1000 time: [14.849 µs 14.857 µs 14.869 µs] change: [−32.666% −32.493% −32.339%] (p = 0.00 < 0.05) Performance has improved. Found 16 outliers among 100 measurements (16.00%) 1 (1.00%) low severe 2 (2.00%) low mild 7 (7.00%) high mild 6 (6.00%) high severe date_trunc_year_nanos_1000 time: [12.385 µs 12.391 µs 12.398 µs] change: [−39.498% −39.449% −39.405%] (p = 0.00 < 0.05) Performance has improved. Found 9 outliers among 100 measurements (9.00%) 2 (2.00%) low mild 4 (4.00%) high mild 3 (3.00%) high severe ``` ## Are there any user-facing changes?

…alExpr::current() (#23532) ## Which issue does this PR close? - Closes #23533. ## Rationale for this change `DynamicFilterPhysicalExpr::current()` is invoked per batch on the RowFilter path (via `PhysicalExpr::evaluate`). It calls `remap_children`, which does a `transform_up` tree walk. For dynamic filters that carry a large `InListExpr` — e.g. `HashJoinExec` pushing down a `col IN build_keys` list of ~150+ values — that walk is dominated by `InListExpr::with_new_children` cloning the whole list on every call. The `inner.generation` only changes when the filter is `update`d — once per HashJoin build, or once per TopK threshold refresh. Between updates every `current()` call recomputes an identical remapped tree. Caching per generation lets per-batch calls short-circuit the walk. ### Measured impact (TPCH SF1, 10 iterations) | | HEAD `pushdown=true` | This PR `pushdown=true` | Baseline `pushdown=false` | |---|---|---|---| | **Q17** | 128 ms | **53 ms** (**2.4× faster**) | 55 ms | | Q18 | 69 ms | 56 ms | 49 ms | | Total (22 queries) | 743 ms | 644 ms (**-13%**) | 531 ms | Q17 (Nation-in-orders subquery) matches the `pushdown=false` baseline after the fix — the DynamicFilter tax is fully eliminated on that shape. Row counts identical across all 22 queries pre/post fix. Profile before the fix: `InListExpr::with_new_children` and its `drop_glue` sat at the top of Q17's samply flamegraph. Part of #20324 (parquet filter-pushdown regression EPIC). ## What changes are included in this PR? - Add `current_cache: Arc<RwLock<Option<(u64, Arc<dyn PhysicalExpr>)>>>` on `DynamicFilterPhysicalExpr` (aliased as `CurrentExprCache` to keep the type readable). - `current()`: read the current `(expr, generation)` from `inner`, return the cached `Arc` if the cached generation matches; otherwise recompute via `remap_children` and store the new pair under a write lock. - Reset semantics: - `update()` bumps `inner.generation`, so the next `current()` misses the cache and recomputes. - `with_new_children` clones the outer struct with new `remapped_children`; the derived filter gets its own fresh cache slot. - `from_parts` (proto deserialization) starts with an empty cache. - No public API changes. ## Are these changes tested? Yes — three new regression tests in `expressions::dynamic_filters::test`: - `test_current_cache_hits_within_generation` — three consecutive `current()` calls return the same `Arc` (pointer-equal) at the same generation, proving the remap walk did not re-run. - `test_current_cache_invalidates_on_update` — after `update()` bumps the generation, `current()` returns a fresh `Arc` (not pointer-equal, and stringly distinct). - `test_current_cache_is_per_derived_filter` — two filters derived via `with_new_children` with different `remapped_children` produce distinct cached `Arc`s and each hits its own cache on subsequent calls. All 21 tests in `expressions::dynamic_filters` (18 existing + 3 new) pass. ## Are there any user-facing changes? No. Internal caching only. No public API changes; behavior is identical modulo the generation-tied fast path.

theirix and others added 6 commits July 14, 2026 10:28

pull Bot locked and limited conversation to collaborators Jul 14, 2026

pull Bot added the ⤵️ pull label Jul 14, 2026

pull Bot merged commit d1e6d45 into TCeason:main Jul 14, 2026
1 check was pending

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[pull] main from apache:main#1337

[pull] main from apache:main#1337
pull[bot] merged 6 commits into
TCeason:mainfrom
apache:main

pull Bot commented Jul 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

pull Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pull Bot commented Jul 14, 2026 •

edited

Loading